Motor function neural control interface for spinal cord injury patients

ABSTRACT

At least one electrical brain signal is received from a patient and is demultiplexed into an efferent motor intention signal and at least one afferent sensory signal (such as an afferent touch sense signal and/or an afferent proprioception signal). A functional electrical stimulation (FES) device is controlled to apply FES to control a paralyzed portion of the patient that is paralyzed due to a spinal cord injury of the patient. The controlling of the FES device is based on at least the efferent motor intention signal. A demultiplexed afferent touch sense signal may be used to control a haptic device. The afferent sensory signal(s) may be used to adjust the FES control.

This application is a continuation of U.S. Application Serial No.16/729,035 filed Dec. 27, 2019 which claims the benefit of provisionalapplication No. 62/798,648 filed Jan. 30, 2019, and which claims thebenefit of provisional application No. 62/787,060 filed Dec. 31, 2018.Provisional application No. 62/798,648 filed Jan. 30, 2019 isincorporated herein by reference in its entirety. Provisionalapplication No. 62/787,060 filed Dec. 31, 2018 is incorporated herein byreference in its entirety.

BACKGROUND

The following relates to the medical arts, neural-control interfacearts, brain-computer interface arts, motor function neural controlinterface arts, functional electrical neuro-motor stimulation arts,neurosensory function assessment and control interface arts, and relatedarts, and to apparatuses and methods employing same for providing motorfunctionality to patients with spinal cord injuries.

Over 100,000 individuals in the United States currently live with acervical spinal cord injury (SCI), in which the cervical spinal cord ispartially or wholly severed. Depending upon where the spinal cord injuryis located, a lower portion of the patient is paralyzed, e.g. afunctionally complete cervical SCI at the C5 level typically results inparalysis of the wrists, hands, and triceps, while a functionallycomplete cervical SCI at the C6 level typically results in paralysis ofthe wrist flexors, triceps, and hands. In a functionally completecervical SCI, motor intention signals for controlling the paralyzedportion of the patient generated by the motor cortex of the brain do notreach the paralyzed portion, and afferent sensory signals from theparalyzed portion are not perceived by the patient.

Various approaches have been developed for assisting such patients. Forexample, voice recognition software and/or eye movement sensors can beused to allow the patient to control a computer or other electronicdevices by voice or eye movements, respectively. However, these patientassistance devices do not replicate the biological motor function lostdue to the paralysis.

Another approach is to employ a Brain-Computer Interface (BCI), alsosometimes referred to as a Neural-Control Interface (NCI), mind-machineinterface (MMI), Direct Neural Interface (DNI), Brain-Machine Interface(BMI), or similar nomenclatures, in conjunction with a FunctionalElectrical Stimulation (FES) device. The BCI “reads” motor intention(Ml) signals generated in the patient’s brain (usually in the motorcortex) using electrodes implanted in the brain or contacting the skinadjacent the brain, and controls the FES device on the basis of the Mlsignals. The FES device includes electrodes implanted in or contactingthe skin of the paralyzed portion of the patient (e.g. an FES cuff foruse with a paralyzed wrist), so that electrical control signalsdelivered by these FES electrodes can electrically stimulate motorfunction. If the Ml signals are accurately assessed as to the intendedmotor function, then the combination of the BCI and the FES device canenable the patient to directly control the paralyzed portion of the bodyby generating motor intentions in the motor cortex. In other words, theBCI/FES device combination bypasses the cervical SCI and “reconnects”the brain with the motor control nerves so that the patient can move theparalyzed portion using the patient’s usual motor intention generated inthe motor cortex. Some examples of such BCI/FES device systems aredescribed in Bouton et al., U.S. Pub. No. 2018/0154132 A1 published Jun.7, 2018 which is incorporated herein by reference in its entirety, andin Bouton et al., U.S. Pub. No. 2018/0154133 A1 published Jun. 7, 2018which is incorporated herein by reference in its entirety.

Although a BCI/FES device system can mimic thought-driven motor control,a difficulty remains in terms of feedback to the patient. In the case ofa functionally complete cervical SCI, the patient cannot perceivesensory signals from the paralyzed portion of the body. In particular,the patient cannot perceive the touch sense due to the functionallycomplete cervical SCI. Typically, the patient is trained to visuallyobserve the motor operation (e.g. gripping an object with a paralyzedhand) and to rely upon seeing the motor operation (e.g. seeing the handgrip the object) to provide feedback as to success (or failure) of thegripping or other motor operation. However, this approach has somedisadvantages. The patient may not visually see the critical point ofcontact, e.g. the fingertips actually touching the object. The visualperception also does not sensitively capture the strength of the touch,e.g. the finger lightly touching the object versus the finger pressingharder against the object versus squeezing the object hard.Additionally, the patient has had long years of developing motor-sensoryfeedback based on the touch sense, and it can be difficult for thepatient to make the transition to relying upon the different approach ofemploying visual feedback in a gripping or other motor operation.

The following describes certain improvements.

BRIEF SUMMARY

In accordance with some illustrative embodiments disclosed herein, anapparatus is disclosed for assisting a patient having a spinal cordinjury. The apparatus comprises an electrical brain signal monitoringinterface, a functional electrical stimulation (FES) device, anelectronic processor, and a non-transitory storage medium. Theelectrical brain signal monitoring interface is configured to record atleast one electrical brain signal of the patient. The FES device isconfigured to connect via FES electrodes with a paralyzed portion of thepatient that is paralyzed due to the spinal cord injury and to controlthe paralyzed portion of the patient by applying FES to the paralyzedportion of the patient via the FES electrodes. The electronic processoris operatively connected with the electrical brain signal monitoringinterface to receive the at least one electrical brain signal and withthe FES device to control the FES applied to the paralyzed portion ofthe patient. The non-transitory storage medium stores instructionsreadable and executable by the electronic processor. The instructionsinclude electrical brain signal demultiplexing instructions that areexecutable by the electronic processor to demultiplex the at least oneelectrical brain signal into an efferent motor intention signal and atleast one afferent sensory signal. The instructions further include FEScontrol instructions that are executable by the electronic processor tocontrol the FES applied to the paralyzed portion of the patient by theFES device based on at least the efferent motor intention signal.

In accordance with some illustrative embodiments disclosed herein, amethod comprises: receiving at least one electrical brain signal from apatient; by an electronic processor, demultiplexing the at least oneelectrical brain signal into an efferent motor intention signal and atleast one afferent sensory signal; and by the electronic processor,controlling a functional electrical stimulation (FES) device to applyFES to control a paralyzed portion of the patient that is paralyzed dueto a spinal cord injury of the patient. The controlling of the FESdevice is based on at least the efferent motor intention signal. Thedemultiplexing may comprise demultiplexing the at least one electricalbrain signal into the efferent motor intention signal and the at leastone afferent sensory signal including at least an afferent touch sensesignal generated by a touch sense of the paralyzed portion of thepatient. In such a case, the method may further comprise, by theelectronic processor, controlling a haptic device worn on a portion ofthe patient where the patient perceives touch sense to deliver a hapticsignal to the patient based on the afferent touch sense signal; and/or,the controlling of the FES device may be based on at least the efferentmotor intention signal and the afferent touch sense signal. Thedemultiplexing may comprise demultiplexing the at least one electricalbrain signal into the efferent motor intention signal and the at leastone afferent sensory signal including at least an afferentproprioception sense signal generated by a proprioception sense of theparalyzed portion of the patient. In such a case, the controlling of theFES device may be based on at least the efferent motor intention signaland the afferent proprioception sense signal. Notwithstanding theforegoing, in some embodiments the controlling of the FES device isbased on only the efferent motor intention signal, and is not based onthe at least one afferent sensory signal.

In the apparatuses and methods of the immediately two precedingparagraphs, the demultiplexing may comprise applying a machine learningcomponent to demultiplex the at least one electrical brain signal intothe efferent motor intention signal and the at least one afferentsensory signal. The instructions may further include instructions to, orthe method may further include, by the electronic processor: trainingthe machine learning component to demultiplex the at least oneelectrical brain signal into the efferent motor intention signal and theat least one afferent sensory signal using labeled training electricalbrain signals which are labeled as to motor intention stimuli and sensestimuli. The machine learning component may, for example, be a supportvector machine (SVM) or an artificial neural network.

These and other non-limiting aspects of the disclosure are moreparticularly discussed below.

BRIEF DESCRIPTION OF THE DRAWINGS

The following is a brief description of the drawings, which arepresented for the purposes of illustrating the exemplary embodimentsdisclosed herein and not for the purposes of limiting the same.

FIG. 1 diagrammatically shows an apparatus for assisting a patienthaving a spinal cord injury.

FIG. 2 diagrammatically shows an illustrative embodiment of the hapticsignal generator of the apparatus of FIG. 1 .

FIG. 3 and FIG. 4 diagrammatically show two illustrative embodiments ofthe functional electric stimulation (FES) signal generator of theapparatus of FIG. 1 .

FIG. 5 diagrammatically shows a training system for training theefferent/afferent demultiplexor of the apparatus of FIG. 1 .

FIGS. 6A, 6B, 6C, 6D and FIGS. 7A and 7B illustrate aspects ofexperiments showing skin stimulation on the arm and hand evokes robustresponses in contralateral primary motor cortex (M1) following cervicalspinal cord injury (SCI) as described herein.

FIGS. 8A, 8B, and 8C illustrate aspects of experiments showing evokedsensory activity in M1 is decodable across skin locations, as describedherein.

FIGS. 9A, 9B, 9C, and 9D illustrate aspects of experiments showing touchcan be decoded from M1 to control closed-loop sensory feedback andenhance hand sensory function, as described herein.

FIGS. 10A and 10B and FIGS. 11A and 11B illustrate aspects ofexperiments showing afferent and efferent M1 activity can besimultaneously decoded to enable ‘sensorimotor demultiplexing’ BCIcontrol and enhancement of sensorimotor function, as described herein.

FIGS. 12A, 12B, and 12C illustrate aspects of experiments showingdecoding of afferent grip intensity levels from M1 activity to enablelimb reanimation regulated by touch.

DETAILED DESCRIPTION

A more complete understanding of the methods and apparatuses disclosedherein can be obtained by reference to the accompanying drawings. Thesefigures are merely schematic representations based on convenience andthe ease of demonstrating the existing art and/or the presentdevelopment, and are, therefore, not intended to indicate relative sizeand dimensions of the assemblies or components thereof.

Although specific terms are used in the following description for thesake of clarity, these terms are intended to refer only to theparticular structure of the embodiments selected for illustration in thedrawings, and are not intended to define or limit the scope of thedisclosure. In the drawings and the following description below, it isto be understood that like numeric designations refer to components oflike function.

As used herein, the “motor cortex” encompasses all components of thesensorimotor cortex, including sub-regions such as those designated asthe primary motor cortex (M1), the primary somatosensory cortex (S1),the premotor cortex, the supplementary motor area (SMA), the posteriorparietal cortex, and so forth. As used herein, the term “brain”encompasses the entire brain of the SCI patient, and hence the term“brain” includes, but is not limited to, the motor cortex.

As used herein, terms such as “efferent motor intention signal”,“afferent sensory signal”, “afferent touch sense signal”, and “afferentproprioception sense signal” are referenced to the brain. Hence,“efferent” denotes a signal output by the brain, or in some embodimentsmore particularly output by the motor cortex of the brain; while,“afferent” denotes a signal received at the brain, or in someembodiments more particularly received by the motor cortex of the brain.For example, an afferent touch sense signal indicates the touch sensesignal received at the brain, as opposed to the touch sense signal inthe finger or other anatomical structure that senses the touch.

As used herein, the term “electrical brain signal monitoring interface”or similar phraseology denotes a device for measuring or monitoring anelectrical brain signal, that is, a signal indicative of electricalactivity of the brain, and in some embodiments more particularlyindicative of electrical activity of the motor cortex of the brain.These terms are to be broadly construed as denoting brain electricalactivity signals in the broadest sense, and are not intended to belimited to detection of any specific type of brain wave or the othernarrower interpretation. As used herein, an electrical brain signalencompasses, for example, a signal acquired by as few as a singleelectrode implanted in the brain or attached to the scalp to detectbrain electrical activity.

As used herein, the term “functional electrical stimulus” or thecorresponding acronym “FES” is intended to indicate any electricalstimulus applied by one or more electrodes that are implanted into a SCIpatient’s tissue or organ or anatomical body part or anatomicalstructure, or attached to skin of such tissue or organ or anatomicalbody part or anatomical structure, so as to electrically stimulatemuscular activity in the issue or organ or anatomical body part oranatomical structure. An FES device is any device including suchelectrodes which is designed to connect with the SCI patient’s tissue ororgan or anatomical body part or anatomical structure. In illustrativeexamples, the FES device is a wrist cuff or sleeve or the like forproviding functional electrical stimulation to an SCI patient’s wristand/or hand and/or forearm. As another example, an FES device may be aleg wrap or leg cuff with electrodes for providing functional electricalstimulation to an SCI patient’s leg and/or foot.

As noted previously, although a BCI/FES device system can mimicthought-driven motor control, a difficulty remains in terms of feedbackto the patient. In the case of a functionally complete SCI, the patientcannot perceive sensory signals from the paralyzed portion of the body.In particular, the patient cannot perceive the touch sense in theparalyzed region due to the functionally complete SCI.

Notwithstanding the foregoing, in approaches disclosed herein, a BCI/FESdevice system includes an electronic processor programmed to demultiplexat least one electrical brain signal (acquired by one or more electrodesimplanted in the brain or attached to the scalp to record brainactivity) into an efferent motor intention signal and at least oneafferent sensory signal. Although a functionally complete SCI preventsthe patient from perceiving sensory signals from the paralyzed portionof the body, this does not mean that afferent sensory signals are notreceived at (or by) the brain. Residual sensory information ispotentially transmitted to the brain of a patient with functionallycomplete SCI, even though the patient does not consciously perceive thesensations. (Usually an SCI is classified as “complete” based onfunction alone, e.g. the patient cannot move or feel below spinal level“X”. Functionally complete SCI does not necessarily imply ananatomically complete SCI, e.g. there may still be a small amount ofspared fibers that reach the brain, and may for example transmit anattenuated touch sense). Moreover, activity in primary motor cortex (M1)and in the primary sensory cortex (S1) may reflect sensorimotorinformation beyond their primary processing designation, e.g. activityin M1 may be modulated by somatosensory feedback of the neuromotor stateof the limb. As shown herein (see Experimental), afferent sensorysignals from a paralyzed region impact the electrical brain signals ofthe motor cortex in spite of the functionally complete SCI, and theseafferent sensory signals are at a sufficiently high level to introducesignificant noise into the extracted motor intention signal. It isexpected that noise due to the afferent sensory signals should besuppressed by demultiplexing the electrical brain signal(s) into anefferent motor intention signal and at least one afferent sensorysignal. The resulting (demultiplexed) efferent motor intention signal isthereby “cleaned up” to provide an improved control signal forcontrolling an FES device to implement the motor intention.

Optionally, the demultiplexed afferent sensory signal(s) may be used forother purposes. In some embodiments, a demultiplexed afferent touchsense signal is used to drive a haptic device worn by the patient on aportion of the body that is not paralyzed due to the SCI. This provideshaptic feedback to the patient based on the patient’s own touch sense,in spite of that touch sense not being perceptible by the patient.

As noted above, by removing the afferent sensory signal(s) by thedemultiplexing, a more “pure” motor intention signal is derived, and insome embodiments this demultiplexed motor intention signal is used byitself for driving the FES device. On the other hand, in some otherembodiments the demultiplexed afferent sensory signal(s) may be used toaugment the demultiplexed motor intention signal for driving the FESdevice. For example, the demultiplexed afferent touch sense signal maybe used as follows. An FES control signal is first generated based onthe (demultiplexed) motor intention signal. This FES control signal iseffective to cause the FES device to control the paralyzed portion ofthe patient to perform a motor action indicated by the motor intentionsignal. Then, the generated FES control signal is adjusted to reduce astrength of the motor action based on a strength of the (demultiplexed)afferent touch sense signal. For example, in a gripping action, themotor intention signal drives the gripping action; however, when theafferent touch sense signal begins to ramp up as the patient’sfingertips begin to touch the object to be gripped, this provides afeedback signal that reduces the strength of the gripping action (e.g.,slows the gripping motion) so as to more gracefully grasp the objectwithout producing too much squeezing pressure. Optionally, this is donein conjunction with the aforementioned optional driving of a hapticdevice in order to concurrently provide the patient with haptic feedbackindicating the hand is contacting the object.

As another example (which may be used with any of the above-describedexamples), the demultiplexed afferent sensory signal(s) may include anafferent proprioception sense signal generated by a proprioception senseof the paralyzed portion of the patient. In some embodiments, the FESapplied to the paralyzed portion of the patient by the FES device iscontrolled based on the motor intention signal and also based on theafferent proprioception sense signal. For example, in a gripping action,the afferent proprioception sense signal may indicate the orientation ofthe palm of the hand. More specifically, demultiplexed afferent signalfor wrist orientation are contemplated to be used to determine whichstimulation pattern to use. The patterns change with wrist orientationdue to anatomical muscle changes. If the current orientation of thewrist is known, it can inform the FES system which parameters to use forthe optimal grip. If the palm is not aligned with the object to begripped, then the FES control may stimulate wrist muscles to rotate thewrist in order to align the palm with the object, prior to (or alongwith) driving the hand to perform the gripping action.

With reference to FIG. 1 , an apparatus for assisting a patient P havinga spinal cord injury (SCI) is diagrammatically shown. (although SCI isthe illustrative example, more generally the disclosed approaches arecontemplated for use in conjunction with other patients with disruptedsensory and motor neurosignal transmission, such as stroke patients withpatients with Parkinson’s disease). An electrical brain signalmonitoring interface 10 is configured to record at least one electricalbrain signal 12 of the patient P via one or more electrodes 14 that areimplanted in the brain of the patient P or attached to the scalp of thepatient P in order to detect electrical brain activity. The electrodescan be invasive (e.g. implanted underneath the scalp) or non-invasive(e.g. placed on the scalp). The electrode(s) 14 may consist of a singleelectrode, or may be an array of electrodes. Preferably, theelectrode(s) 14 are positioned in order to detect brain activity fromthe motor cortex of the brain, as motor intention signals can begenerated in the motor cortex. In the case of a single electrodeimplanted in the brain or attached to the scalp, the single electrode ispreferably positioned to detect brain activity in the primary motorcortex (M1). In the case of an array of electrodes, variousconfigurations are contemplated. In one approach, the array ofelectrodes are arranged to span the entire motor cortex and may extendbeyond the area of the motor cortex. In another approach, the array ofelectrodes is arranged entirely within the motor cortex area. In someembodiments, the placement of the electrode(s) 14 is chosen prior toimplanting the electrodes by fusing functional magnetic resonanceimaging (fMRI) activation maps obtained while the patient attemptedmovements of the paralyzed anatomy (so as to generate motor intentionsignals in the motor cortex) co-registered to the preoperative planningMRI. In some actually conducted experiments (see Experimental), theelectrode(s) 14 were a Utah 96 channel microelectrode array (BlackrockMicrosystems, Inc., Salt Lake, Utah) implanted in the left primary motorcortex of the patient P. These are merely illustrative examples.

The electrical brain signal monitoring interface 10 may be any devicethat is capable of measuring brain electrical activity detected by theelectrode(s) 14. (In some alternative embodiments, it is contemplated toperform this monitoring via peripheral nerve monitoring, e.g. usingelectrode(s) reading neurological signals conducted along the spinalcord). This may be a commercial electrical brain signal monitoringinterface device of a type designed to measure brain waves, or may beanother type of neural recording system. The brain electrical activitythat is measured may include electrical signals produced by neuralactivity in the nervous system such as action potentials, multi-unitactivity, and local field potential.

In the actually conducted experiments (see Experimental), the electricalbrain signal monitoring interface 10 was a Neuroport neural dataacquisition system which recorded data from all 96 channels of the Utah96 channel microelectrode array, sampled at 30 kHz and band passfiltered from 0.3-7.5 kHz using a third order Butterworth analoghardware filter, then digitized and sent to a computer 18 for storageand/or further processing using an interface constructed in MATLAB 2014a(The MathWorks; Natick, MA). Again, these are merely illustrativeexamples.

The apparatus further includes a functional electrical stimulation (FES)device 20. The FES device contains a set of electrodes that are used toprovide electrical stimulation. Very broadly, the term “sleeve” is usedto refer to a FES device with a structure that surrounds a body part,for example, an arm, leg, or torso. The sleeve can take the form of ashirt, or pants, if desired. The FES device can also contain sensors formonitoring movement of the body part (position, orientation,acceleration, etc.), which can be used to track the movement of the bodypart. In the illustrative example and in the actually conductedexperiments, the FES device was a multi-channel stimulator and aflexible cuff with up to 130 electrodes that is wrapped around theparalyzed forearm of the patient P, with hydrogel disks (Axelgaard,Fallbrook, CA) placed between the electrodes and skin to act as aconduction enhancer. The electrodes in the actually conductedexperiments were 12 mm in diameter and were spaced at 22 mm intervalsalong the longitudinal axis of the forearm and 15 mm intervals in thetransverse direction. Current-controlled, monophasic rectangular pulses(50 Hz pulse rate and 500 µs pulse width) were used to provideelectrical stimulation. Pulse amplitudes ranged from 0 to 20 mA and wereupdated every 100 ms. Stimulator calibrations were performed for eachmovement using an anatomy-based trial-and-error method to determineappropriate electrode spatial patterns.

It will be appreciated that the FES device is preferably designed forthe particular paralyzed anatomy to be stimulated (e.g., a leg wrap orcuff would be suitable for driving a paralyzed leg), and may alsooptionally be designed and fitted for the individual patient. Somefurther non-limiting illustrative examples of FES devices are describedin Bouton et al., U.S. Pub. No. 2018/0154132 A1 published Jun. 7, 2018which is incorporated herein by reference in its entirety, and in Boutonet al., U.S. Pub. No. 2018/0154133 A1 published Jun. 7, 2018 which isincorporated herein by reference in its entirety.

The right side of FIG. 1 diagrammatically shows processing operationsperformed on the electrical brain signal(s) 12 captured by theelectrical brain signal monitoring interface 10 in order to generate FEScontrol signal for controlling the FES device 20. As diagrammaticallyshown, the processing operations include applying an efferent/afferentdemultiplexor 22 to demultiplex the electrical brain signal(s) 12 intoan efferent motor intention (Ml) signal 24 and at least one afferentsensory signal 26, 28. In the illustrative example, the afferent sensorysignals include an afferent touch sense (TS) signal 26 generated by atouch sense of the paralyzed portion of the patient (e.g., the paralyzedhand of the patient P in the illustrative example), and an afferentproprioception sense signal 28 generated by a proprioception sense ofthe paralyzed portion of the patient. In illustrative examples, theefferent/afferent demultiplexor 22 comprises a machine learningcomponent that is applied to demultiplex the electrical brain signal(s)12 into the efferent motor intention signal 24 and the at least oneafferent sensory signal 26, 28. Advantageously, the demultiplexor 22 candemultiplex motor intention, touch, and proprioception allsimultaneously, and can also demultiplex them separately in time. Themachine learning component is trained to demultiplex the electricalbrain signal(s) 12 into the efferent motor intention signal 24 and theat least one afferent sensory signal 26, 28 using labeled trainingelectrical brain signals which are labeled as to motor intention stimuliand sense stimuli (see FIG. 5 ), and the particular afferent sensorysignal(s) that is/are demultiplexed by the demultiplexor 22 depends uponwhich afferent sensory signal(s) are labeled and whose demultiplexing istrained using such labels. In some embodiments, the demultiplexor 22 isonly trained to demultiplex the afferent touch sense (TS) signal 26 butnot the afferent proprioception sense signal 28 (or, in otherembodiments, vice versa). Because the efferent/afferent demultiplexor 22operates on the electrical brain signal(s) 12, the afferent sensorysignal(s) 26, 28 output by the demultiplexor 22 are afferent signalsreceived at (or by) the brain (and more particularly at or by the motorcortex, if the electrode(s) 14 are positioned to read brain activity ofthe motor cortex). As discussed previously, these afferent sensorysignals are generally not perceived by the patient P due to thefunctionally complete SCI - but they are present at a subconscious levelin at least some SCI patients. (It will be appreciated that the approachwill not work if the patient’s functionally complete SCI is such that nosensory signals from the paralyzed body part reach the brain, but thisis not the case in at least some SCI patients, even with nominally“complete” SCI).

Although not illustrated, it is also contemplated that additionalafferent sensory signals may be demultiplexed, such as visual sensorysignals to the extent that such may be present in the motor cortex.

As further diagrammatically shown in FIG. 1 , the efferent motorintention (Ml) signal 24 is input to an FES control signal generator 30to generate control signals for controlling the FES device 20 to drivethe paralyzed body portion (e.g., the hand and possibly also the wristof the patient P in the illustrative example) to perform the actionindicated by the motor intention signal 24. Because the demultiplexor 22has separated the motor intention signal 24 from the afferent sensorysignal(s) 26, 28, the motor intention signal 24 is less noisy (since theafferent sensory signal(s) 26, 28 constitute noise superimposed on themotor intention signal generated in the motor cortex). Optionally, theafferent sensory signal(s) 26, 28 may be used in other ways. Forexample, the afferent touch sense signal 26 may be used to reduce thegripping force (see FIG. 3 ), and/or the afferent proprioception sensesignal 28 may be used to align the palm of the hand with an object to begripped (see FIG. 4 ) and/or to change stimulation parameters to accountfor anatomical musculature changes during pronation/supination. Asfurther illustrated in FIG. 1 , the afferent touch sense signal 26 mayoptionally also be used by a haptic device drive signal generator 32 todrive a haptic device 34 worn by the patient on a portion of the bodythat is not paralyzed by the functionally complete SCI. (In theillustrative example of a functionally complete SCI at the C5/C6 level,the hand and wrist are typically paralyzed, but the upper arm where theillustrative haptic band 34 is worn is usually not paralyzed and thepatient P can sense the vibration (i.e. the haptic signal) produced bythe haptic band 34 worn on the upper arm). While a haptic device 34 isused in the illustrative example, other devices for producing ahuman-perceptible feedback signal indicating the strength of theafferent touch sense signal 26 are contemplated, such as displaying thestrength as a length of a bar shown on a computer display 36 or soforth.

The various signal/data processing 22, 30, 32 is suitably performed byan electronic processor, such as the computer 18 (as in the actuallyconducted experiments) or a cellular telephone (cellphone) 38, and/or soforth. Employing the cellphone 38 as the electronic processoradvantageously could facilitate patient mobility, e.g. the cellphone 38may be carried in a holster or the like. In the actually conductedexperiments, the electrical brain signals 12 were carried by a wirebundle 40 to the computer 18, and wiring 42 was provided fordistributing the FES control signals for the FES electrodes of the FESdevice 20 and the haptic control signal to the haptic device 34. In thecase of using the cellphone 38, wireless communication of the electricalbrain signal(s) 12 and FES and haptic control signals may be preferable,e.g. using Bluetooth™ or another short-range wireless communicationprotocol.

As is known in the computer arts, to configure the electronic processor(e.g. computer 18 or cellphone 38) to perform the described tasks,suitable software is employed to program the electronic processor 18, 38to perform these tasks. Hence, a non-transitory storage medium (notshown) stores instructions that are readable and executable by theelectronic processor 18, 38 to perform these tasks. The non-transitorystorage medium may, for example, include a Read-Only Memory (ROM), flashmemory, or other electronic storage (e.g., the internal memory of thecellphone 38); a hard disk or other magnetic storage (e.g. of thecomputer 18); an optical disk or other optical storage; variouscombinations thereof, and/or so forth. The instructions may (by way ofnon-limiting illustrative example) include: electrical brain signaldemultiplexing instructions that are executable by the electronicprocessor 18, 38 to demultiplex the electrical brain signal(s) 12 intothe efferent motor intention signal 24 and the at least one afferentsensory signal 26, 28; FES control instructions that are executable bythe electronic processor 18, 38 to control the FES applied to theparalyzed portion of the patient by the FES device 20 based on at leastthe motor intention signal 24; touch sensory feedback instructions thatare executable by the electronic processor 18, 38 to control the hapticsignal delivered to the patient via the haptic device 34 based on theafferent touch sense signal 26; and/or so forth.

The efferent/afferent demultiplexor 22 is trained to perform thedemultiplexing using labeled data. It is contemplated that this trainingmay be done once for all patients. However, due to the individualizednature of brain neural activity, it is expected that better FES controlperformance will be achieved by individually training theefferent/afferent demultiplexor 22 for the individual patient P withwhich the efferent/afferent demultiplexor 22 is to be used. (This may beab initio training, or may be a type of update training, i.e. tuning, ofa machine learning component that is initially/previously trained onpopulation data, that is, labeled data acquired for a population ofpatients with similar SCI). Hence, the instructions stored on thenon-transitory storage medium may further include machine learninginstructions that are executable by the electronic processor 18, 38 totrain the machine learning component (implementing the efferent/afferentdemultiplexor 22) to demultiplex the electrical brain signal(s) 12 intothe efferent motor intention signal 24 and the at least one afferentsensory signal 26, 28 using labeled training electrical brain signals(of the individual patient P in the case of individualized training)which are labeled as to motor intention stimuli and sense stimuli.

With reference now to FIG. 2 , an illustrative example of the optionalhaptic feedback aspect is described in further detail. The afferenttouch sense signal 26 is amplified by a nonlinear (or, in anotherembodiment, a linear) amplifier 50, and the amplified output is input toa haptic device driver 52 to generate a haptic signal 54 for driving thehaptic device 34. The gain, offset, and/or other parameters of thenonliniear amplifier 50 may be adjusted by trial-and-error to tune thehaptic feedback to a level that the patient P finds to be most helpful.

With reference now to FIG. 3 , an illustrative example of the optionalafferent touch sense feedback in adjusting the FES control is described.The illustrative example employs a FES signal generator 60 of a typeused in FES control employing only the motor intention signal (that is,without any efferent/afferent demultiplexing). Thereafter, the afferenttouch sense signal 26 is used in an operation 62 to reduce the FESsignal, that is, to reduce the strength of the illustrative grippingaction, in response to an increasing magnitude of the afferent touchsense signal. The operation 62 may, for example, apply no reduction solong as the afferent touch sense signal 26 is below some threshold(corresponding to some percentage over the root-mean-square noise levelof the afferent touch sense signal 26, for example). When the afferenttouch sense signal 26 increases above this threshold (indicating thefingers are beginning to touch the object to be gripped) the operation62 begins to reduce the rate of closure of the hand being driven toperform the gripping operation. As the afferent touch sense signal 26continues to increase, the operation 62 further reduces the rate ofclosure of the hand being driven to perform the gripping operation. Whenthe afferent touch sense signal 26 increases above an upper threshold(indicating the hand has fully grasped the object) then the operation 62reduces the rate of closure of the hand to zero and causes it tomaintain its current grip force. This is merely an illustrative example,and other types of feedback control of FES-driven actions may beemployed depending upon the type of action and the type of availableafferent touch sense signal. The FES cuff driver 64 then generates theFES cuff control signals 66 for driving the FES device 20 (see FIG. 1 )in accordance with the generated and reduced FES signal.

With reference now to FIG. 4 , an illustrative example of the optionalafferent proprioception sense feedback in adjusting the FES control isdescribed. The illustrative example again employs the FES signalgenerator 60 already described with reference to FIG. 3 . At a decisionoperation 70, the afferent proprioception sense signal 28 is used toassess whether the palm of the hand is oriented correctly for graspingthe object. For example, if the object is a can or glass containing adrink, then the palm should be oriented vertically in order to grasp thesides of the can or glass. If the palm is determined to be orientedproperly at decision 70, then FES cuff driver 64 generates the FES cuffcontrol signals 66 for driving the FES device 20 as already describedwith reference to FIG. 3 . On the other hand, if the palm is notcorrectly oriented (e.g. palm facing up, or palm facing down) then in anoperation 72 an additional FES signal component is added in order tocause the FES device 20 to rotate the wrist into position for graspingthe can or glass, and this additional wrist drive FES signal is alsooutput by the FES cuff driver 64. Additionally or alternatively, thedemultiplexed afferent signals may be used to adjust stimulationparameters based on wrist orientation.

It should be appreciated in understanding the flow diagrams of FIGS. 2-4that the processing in the dashed boxes is performed by the hapticsignal generator 32 (for FIG. 2 ) or the FES signal generator 30 (forFIGS. 3-4 ), and that the processing diagrammatically shown in FIGS. 2-4is performed in a continuing loop, so that for example the FES signalreduction 62 of FIG. 3 may be initially zero for the first iterations,until the continued FES-driven closure of the hand causes the fingers tobegin contacting the object whereupon later iterations of the FES signalreduction 62 begin to reduce the closure rate of the hand.

With reference to FIG. 5 , the training of the machine learningcomponent that implements the efferent/afferent demultiplexor 22 isdescribed. The machine learning component may, by way of non-limitingillustration, be a support vector machine (SVM, as in the actuallyconducted experiments), an artificial neural network (e.g. a deep neuralnetwork, DNN, a convolutional neural network, CNN, or so forth), orother type of machine learning system. The parameters adjusted by thetraining depend upon the type of machine learning component employed.For example, in an SVM the parameters that are adjusted during thetraining include the hyperplane parameters, e.g. parameters of a vectorw oriented normal to the hyperplane, and a plane offset b. In the caseof an artificial neural network, the parameters that are adjusted duringthe training are typically parameters (e.g. weights, activation values,et cetera) of the neurons of the various layers of the artificial neuralnetwork. The training adjusts the parameters of the machine learningcomponent so that when the machine learning component processes labeledtraining data the outputs (in the instant case, the efferent motorintention signal 24 and the afferent sensory signal(s) 26, 28) optimallymatch ground-truth labels provided with the training data. To provideindividualized responses, the training data may comprise electricalbrain signal(s) versus time 80 measured for the individual patient P forwhich the demultiplexor 22 is being trained. In other words, thetraining data 80 corresponds to the electrical brain signal(s) 12 ofFIG. 1 measured as a function of time (i.e. a data stream of timestampedelectrical brain signal samples. The ground truth labels 82 are themotor intention stimuli (that is, what the patient P intended to do withthe paralyzed body part at a given time) and the sense stimuli. Thisinformation can be obtained in various ways, e.g. high speed videoand/or force sensor(s) 84 may be provided in conjunction with manualand/or automatic labeling 86. Some illustrative labeling examplesfollow.

In one approach, force sensors are disposed on the fingers of thepatient P. When the patient grips an object using the BCI/FES devicesystem, these force sensors detect when the fingers touch the objectbeing gripped, and the data stream of timestamped force sensor signalsamples are aligned in time with the data stream 80 of electrical brainsignal samples to identify when a touch stimulus occurs. This approachis well suited for automated generation of touch sense labels. For theproprioception sense signal, it is contemplated to employ a magnetometerand/or accelerometer mounted to the hand to measure its orientation inreal time, again facilitating automated generation of proprioceptionsense labels (e.g. palm up/down/sideways/specific angle/etc).

In another approach, high speed video is acquired as the patient Pperforms an action using the paralyzed body part driven by the BCI/FESdevice system. The high speed video is acquired as the patient P gripsan object using the BCI/FES device system. The video frames aretimestamped and can be retrospectively analyzed to determine when thefingers began touching the object being gripped, and/or to determine thepalm orientation, and/or so forth. This could be done automatically withsuitable (complex) video image analysis, or may be done manually.

The motor intention labels can be similarly obtained from video oranother timestamped input. For example, the patient P can be instructedto grip an object when a light within the patient’s field of visioncomes on, and the ground truth motor intention can then be set to thetime when the light comes on (as measured by a timestamped controlsignal of the light, or by timestamped video frames that capture thelight). To account for any time lag between the patient seeing the lightcome on and actually initiating the motor intention in the motor cortex,the training can include a single delay parameter that is optimizedalong with the machine learning component.

With the training data 80 collected and a corresponding time stream ofground truth labels 82 provided, a training component 88 consumes thistraining dataset 80, 82 to train the SVM, artificial neural network, orother machine learning component that implements the efferent/afferentdemultiplexor 22. The training component 88 can employ any suitabletraining algorithm, e.g. backpropagation in the case of artificialneural network training. The training algorithm(s) applied are chosenbased on the type of machine learning component (e.g. SVM or particulartype of artificial neural network). As is known in the machine learningarts, some training data may be kept aside and used as validation datato test the trained efferent/afferent demultiplexor 22.

Experimental

In the following, some actually conducted experiments demonstratingvarious aspects of the methods and apparatuses disclosed herein aredescribed. These experiments generally employ the system depicted inFIG. 1 .

A study was conducted of sensorimotor neural signal demultiplexing, toenable a BCI system capable of simultaneously controlling multipleassistive devices for restoring both motor and sensory function. Allexperiments were performed in a chronically paralyzed participant with aC5/C6 SCI. Experiments first assessed the participant’s residual handsensory function. He was unable to perceive sensory stimuli to skininnervated below spinal level C6. This sensory impairment was alsopresent during FES-mediated object grip. For example, the participantoperated below chance when asked to report if he was gripping an objectin the absence of visual feedback, a significant sensory impairmentfurther contributing to motor dysfunction.

Experiments next investigated whether this residual sensory informationcould significantly modulate neural activity following skin stimulation.Sensory stimuli robustly modulated contralateral M1. Stimulation of skininnervated from above or at the C5/C6 SCI evoked time-locked neuralmodulation, lasting ~10 times longer than the stimulus duration (FIG.6B) (FIG. 6C: F[3,24] = 7.6, p < 0.001; FIG. 6D: F[3,24] = 8.7, p <0.001). Stimuli to skin innervated from below the SCI (index and middle)evoked modest neural modulation in M1. As expected, separate controlexperiments revealed no M1 activation following sensory stimuli to theleft arm, ipsilateral to the M1 recording array. In lower frequencies ofneural activity (5-20 Hz), sensory responses also propagated across thearray away from primary somatosensory cortex (S1) via a traveling wavefront. Sensory stimuli also led to a ~2-fold increase in traveling waveprobability, only for semi-intact sensory circuits (forearm & thumb,FIG. 7B; F[3,48]=9.1, p<0.001). These results show that sensory stimulito skin innervated from both above and below the SCI significantlymodulates M1 in the patient.

Next, experiments explored whether this sensory activity can be decodedfrom M1. Decodable sensory events could control a feedback device forimproving the impaired sense of touch and subsequent upper limbsensorimotor function. A support vector machine (SVM) was trained todetect the skin region being passively stimulated (i.e., a ‘passivesensory decoder’), given the underlying neural activity (FIG. 8A).Sensory stimulus location was reliably decoded from M1 across a periodof several months, performing significantly above chance with low falsepositive rates (FIGS. 8B and 8C). Interestingly, decoders for locationsthat the participant can feel (forearm and thumb) performed equivalentlyto decoders for locations that the participant largely cannot feel(index and middle). This result demonstrates the ability to decoderesidual sensory neural activity that is below conscious perception,from functionally relevant hand dermatomes.

Residual sensory activity could also be decoded using in a morechallenging context during active object touch using a separate SVM(i.e., a ‘touch decoder’; FIG. 9 ). During validation experiments, touchdecoders were time locked to force application from the hand andperformed with high responsiveness during object touch events (~85 %;FIG. 9A, ‘Touch’; F[3,68] = 209, p < 0.0001). FES alone or movementalone were control cues lacked object touch events, and did not activatethe touch decoder, as expected (FIG. 9A, ‘No Touch’). These resultsreveal that residual touch neural activity can be reliably decoded fromM1 during active object manipulation.

The participant was next interfaced with a vibrotactile array 34 on theright bicep (corresponding to the haptic device 34 of FIG. 1 ), toenable closed-loop sensory feedback (FIG. 9B; also showing an embodimentof the FES device 20 of FIG. 1 ). The vibrotactile array 34 wascontrolled in real time by a touch decoder, to enhance the perception ofhand sensory events that are significantly impaired following SCI. Theparticipant improved from a C5/C6 to a C8/T1 level of sensory perceptionusing the closed-loop sensory feedback interface, enabling improvedobject touch recognition from below chance to an over 90% recognitionrate (FIG. 9C, righthand bar; t(20) = 2.5, p = 0.02; FIG. 3D) comparedto control (FIG. 9C, lefthand bar). These significant sensoryimprovements were driven by sub-perceptual sensory neural activity thatis demultiplexed from M1 and transformed into conscious perception.

A still further set of experiments were performed to demonstrate thatafferent and efferent activity in M1 can be simultaneously demultiplexedto control multiple devices, constituting a ‘sensorimotordemultiplexing’ BCI (FIG. 1 ). During a modified grasp and release test(GRT), the participant was first cued to position his hand around theobject (FIG. 10A, cue at 0 s), and then generate motor intention toactivate FES to grasp and transfer the object (FIG. 10A, cue at 2 s).The object touch decoder always preceded the motor intention decoder(FIGS. 10A and 10B), and was time locked to object touch as observed invideo.

In some experiments, ‘sensorimotor demultiplexing’ BCI control wasfurther enabled using the simultaneous decoding of touch and motorintention during a set of upper limb assessments. This closed-loopdemultiplexing BCI system enabled significant improvements in sense ofagency (FIG. 11A; t(46) = 3, p = 0.004), movement decoder latency (FIG.11B, left side; t(148) = 2.9, p = 0.004), and GRT motor performancemetrics (FIG. 11B, right side), compared to a motor-only BCI control.These results provide substantial evidence that sensory feedback duringmovement can enhance sense of agency and several other sensorimotorfunctions in patients with upper limb dysfunction. These findingsdemonstrate a BCI system that simultaneously demultiplexes afferent andefferent activity from human cortex to activate multiple assistivedevices and enhance function.

The experiments demonstrate the ability to reanimate both motor andsensory function in a paralyzed and largely insensate limb. There arealternative ways to provide sensory feedback, including intracorticalmicrosimulation (ICMS) in S1. Tactile-based feedback enables rapidsensory perception, significantly faster compared to artificial ICMS inS1. In the reported experiments it was chosen to use the participant’snatural remaining sensory circuitry for touch decoding and address theneed of SCI patients to use their own hand during upper limb activity.

BCIs provide a way to treat patients suffering from and array offunctional deficits. Accurately and consistently decoding a controlsignal for a single assistive device is a significant challenge forBCIs. Embodiments disclosed herein extend capabilities of BCI technologyto simultaneously decipher multiplexed afferent and efferent activity inM1 and control multiple devices simultaneously. Closed-loop sensoryfeedback also improved cognitive aspects of movement ownership, which isuseful for sensorimotor performance. The experimental results show thatsub-perceptual residual neural information can be reliably decoded fromthe human brain, and transformed to conscious perception to augmentfunction.

The human cortex is generally modular and can encode a variety ofstimuli or other activity. The sensory signal utilized in experimentsreported herein may arrive in M1 directly, or from a separate source.Furthermore, evidence is accumulating that M1, and several othercortical modules, encode a multiplicity of features related toexperience beyond their primary processing designation. In BCIapplications contemplated herein, an array of control signals canpotentially be demultiplexed from a single recording site, or multipledistributed interfaces. Advanced decoding strategies may be employed todecipher the multitude of representations encoded in neural activity andenabling demultiplexing BCIs. Regardless, the experimental resultsdescribed herein present progress towards the design of next-generationhuman-machine interfaces capable of demultiplexing multimodal neuralinformation for distributed device control and functional improvement.

With particular reference to FIGS. 6A through 6D, aspects of experimentson skin stimulation on the arm and hand evoking robust responses incontralateral primary motor cortex (M1) following cervical spinal cordinjury (SCI) are illustrated. In FIG. 6A, a three-dimensionalreconstruction of the participant’s cerebrum using T1 magnetic resonanceimaging (MRI) images is shown. A box 100 depicts the microelectroderecording array implanted in left M1 (S1 = primary somatosensory cortex;PMC = premotor cortex). In FIG. 6B, peri-stimulus time histograms wereused to assess neural modulation in M1 (skin stimulation occurs at time0, vertical dashed line). Stimulation of the forearm or thumb evokedtime locked multi-unit activation, with smaller neural responses fromindex or middle. As seen in FIG. 6C, stimulation of the thumb evoked thelargest response magnitude. As seen in FIG. 6D, stimulation of the thumbevoked the shortest response latency (* = p < 0.05; ** = p < 0.01; # =different from Forearm at p < 0.05).

With particular reference to FIGS. 7A and 7B, skin stimulation alsoevoked traveling waves of activity propagating from S1 (left side ofarray) to PMC (right side of array) (5-20 Hz activity. FIG. 7A presentsexemplar traveling wave front; color coded local field potential, µV).FIG. 7B presents traveling wave probability which significantlyincreased only following stimulation to semi-intact skin locations (** =p< 0.01). These results illustrate that somatosensory stimuli evokerobust neural modulation in contralateral M1 following cervical SCI(data in FIGS. 6B, 6C, 6D, 7A, and 7B is for the maximum stimulationintensity). Data presented are mean ± S.E.M.

With particular reference to FIGS. 8A through 8C, aspects of experimentsshowing evoked sensory activity in M1 is decodable across skinlocationsare illustrated. FIG. 8A presents average color coded map ofmean wavelet power (MWP) across all recording channels before, during,and after a sensory stimulus (MWP map for all Forearm stimuli; stimulusoccurs at time 0). Support vector machine (SVM) decoders were builtusing MWP inputs from a recording at a given skin stimulation location(forearm, thumb, index, or middle) or for a rest period. These decodersreliably classified sensory stimulus location, demonstrating significantsensitivity above chance level with low false positive rates for boththe maximum (as shown in FIG. 8B) and minimum stimulation intensities(as shown in FIG. 8C) (the confusion matrices of FIGS. 8B and 8C showcolor coded decoder values; * = significantly above chance at p<0.001).These results illustrate that sensory stimulus location can bedemultiplexed from neural activity in M1.

With particular reference to FIGS. 9A through 9D, aspects of experimentsshowing touch can be decoded from M1 to control closed-loop sensoryfeedback and enhance hand sensory function are illustrated. Theseexperiments used neural data to construct a SVM touch decoder. Duringneural recordings, the participant systemically placed his hand onto astandardized can object, exposing the palmar hand regions to activeobject touch. With reference to FIG. 9A, SVM touch decoders were nextvalidated, involving touch or no touch cues. Touch decoders respondedwith significantly higher responsiveness during object touch events(leftmost two bars of FIG. 9A), compared to FES alone (third bar fromthe right) or movement alone (rightmost bar) events lacking objecttouch. These results illustrate that machine learning algorithms can betrained on M1 activity to accurately demultiplex active object touch inM1 with minimal contribution from other sources. As diagrammaticallyshown in FIG. 9B, touch decoders next controlled closed-loop feedbackvia a vibrotactile array 34 around the sensate bicep skin. Closed-loophaptic feedback triggered by residual sensory information in M1 enhancedthe participant’s level of sensory perception from C5/C6 to C8/T1, andmore than doubled object touch recognition during object grip (as seenin FIG. 9C, up to ~90%) (* = p < 0.05). FIG. 9D presents exemplarycolor-coded mean wavelet power (MWP) input (top) and SVM decoder outputs(bottom) during the object touch recognition assessment (object placedon cues (vertical lines in the decoder output plot) 2, 4, & 6). Theseresults demonstrate that residual sub-perceptual sensory neuralinformation can be demultiplexed in M1 to trigger closed-loop feedbackand significantly improve sensory function. Data presented are mean ±S.E.M.

With particular reference to FIGS. 10A and 10B and FIGS. 11A and 11B,aspects of experiments showing afferent and efferent M1 activity can besimultaneously decoded to enable ‘sensorimotor demultiplexing’ BCIcontrol and enhancement of sensorimotor function are illustrated. Theseexperiments employed the experimental setup described previously withreference to FIG. 1 to perform a modified grasp and release test (GRT)task with the ‘sensorimotor demultiplexing’ BCI. Support vector machineswere first trained to demultiplex afferent touch signaling (touchdecoder) and efferent motor intention (motor decoder) from M1 activity.With reference to FIG. 10A, the touch decoder described herein withreference to FIGS. 9A through 9D was first challenged with a competingmotor intention decoder (during a modified GRT). Touch decoders weretriggered before motor decoders on all object transfers (time 0 = startof cue). With reference to FIG. 10B, the touch decoder led the motordecoder by an average of 1.79 ± 0.11 s. These results illustrate thatafferent touch and efferent movement intention can be simultaneouslydemultiplexed from M1.

The touch decoder was next used to control closed-loop sensory feedbackhaptic device 34 (see FIG. 1 ) and enhance hand touch events. The motordecoder was used to control the FES device 20 of the arm (see FIG. 1 )to produce hand movement. With reference to FIGS. 11A and 11B,closed-loop sensory feedback triggered by demultiplexed touch neuralactivity significantly improved the participant’s sense of agency (FIG.11A), movement decoder latency (FIG. 11B, left side), and GRT motorperformance metrics (FIG. 11B, right side). These results demonstratethe ability to simultaneously decode afferent and efferent informationfrom M1 and activate multiple assistive devices for augmentingsensorimotor function, constituting a ‘sensorimotor demultiplexing’ BCI(** = p < 0.01). Data presented are mean ± S.E.M.

With particular reference to FIGS. 12A, 12B, and 12C, experiments aredescribed which investigate decoding of afferent grip intensity levelsfrom M1 activity to enable limb reanimation regulated by touch. Theseexperiments address the burden on BCI users to control moment-to-momentmovement using a constant decoded motor intention. Instead of constantmotor intention, touch intensity signaling alone can potentiallyregulate limb reanimation during object grip. This capability couldensure appropriate BCI-mediated grip force application, while alsofreeing the BCI user’s attention and visual stream for other importantactivities. Experiments were first performed to assess the hypothesisthat multiple levels of afferent touch intensity signaling from the handcould be decoded from M1 activity. Grip force output was measured usinga replicate of the standardized object with built in calibrated forcesensors. Three different levels of afferent touch and grip intensitysignals could be reliably decoded from M1 activity during object gripevents with low false positive rates (overall ~87 % responsiveness; FIG.12A). As expected, touch and grip intensity decoders were not activatedduring control cues which did not have touch and grip events (see the“FES Alone” entries in FIG. 12A). Therefore, multiple levels of touchand grip intensity could be decoded from M1 activity.

As a proof of concept demonstration, the participant next enabled limbreanimation regulated by decoded afferent touch and grip intensityactivity in M1. Trials were initiated at a high grip force (FIG. 12B,“Touch & High Grip” trials) or a low grip force (FIG. 12C, “Touch & LowGrip” trials) during a long duration grip of the object. Decoded “Touch& High Grip” served to decrease FES and grip force. Decoded “Touch & LowGrip” served to increase FES and grip force. During “Touch & High Grip”trials (FIG. 12B), grip force gradually decreased by a total of 810grams (slope test: F = 29, p = 0.003). This was mediated by adecoder-controlled decrease in FES of 1.3 mA. During “Touch & Low Grip”trials (FIG. 12C), grip force gradually increased by a total of 120grams (slope test: F = 19, p = 0.007). This was mediated by adecoder-controlled increase in FES of 0.4 mA. As expected, touchregulated grip forces initially exhibited an adjustment period followedby a steady state equilibration.

These results support the hypothesis that appropriate grip forceregulation can be controlled by decoded afferent touch and gripintensity activity in M1. These results extend the “sensorimotordemultiplexing” BCI control results. “Touch regulated grip intensity”BCI control can be used to enable automated movement cascades, whilesimultaneously addressing the desire of patients with SCI to use theirown hand. Overall, these results suggest that sensory discompletenesscan be leveraged for multimodal restoration of touch and motor function.

Further details of the above-describe experiments are as follows. Theparticipant was a 27-year-old male with a cervical SCI at C5/C6. Aseries of experiments were performed using either passive sensorystimulation (FIGS. 6A-6D, FIGS. 7A and B, and FIGS. 8A-C) or activeobject manipulation (FIGS. 9A-D, FIGS. 10A and 10B, FIGS. 11A and 11B,and FIGS. 12A, 12B, and 12C) to assess cortical neurophysiology, neuralsignal decoding, and assistive device control for upper limb functionalimprovement. All methods are separated into either passive sensorystimulation or active object manipulation experiments. All experimentswere performed across a total of approximately 1 year. Approval for thisstudy was obtained from the U.S. Food and Drug Administration(Investigational Device Exemption) and the Ohio State University MedicalCenter Institutional Review Board (Columbus, Ohio). The study metinstitutional requirements for the conduct of human subjects and wasregistered on the ClinicalTrials.gov website (Identifier NCT01997125;Date: Nov. 22, 2013). All experiments were performed in accordance withthe relevant guidelines and regulations set by the Ohio State UniversityMedical Center. The participant referenced in this work providedpermission for photographs and videos and completed an informed consentprocess prior to commencement of the study.

As previously noted, the study participant was a 27-year-old male withstable, non-spastic C5/C6 quadriplegia from cervical SCI. He underwentimplantation of a Utah 96 channel microelectrode array (BlackrockMicrosystems, Inc., Salt Lake, Utah) in his left primary motor cortex.The hand area of motor cortex was identified preoperatively by fusingfunctional magnetic resonance imaging (fMRI) activation maps obtainedwhile the patient attempted movements co-registered to the preoperativeplanning MRI. Neural data was acquired using a Utah microelectrode array(Blackrock Microsystems, Inc., Salt Lake City, Utah) and the Neuroportneural data acquisition system. Recorded data from all 96 array channelswas sampled at 30 kHz and band pass filtered online from 0.3 - 7.5 kHzusing a third order Butterworth analog hardware filter. The neural datawas then digitized and sent to a PC for saving or further on-lineprocessing using a custom interface in MATLAB 2014a (The MathWorks;Natick, MA).

Neural signal conditioning and decoding using SVMs was done as follows.We used stimulation artifact removal, mean wavelet power (MWP)estimation, and non-linear SVM. FES-induced stimulation artifacts weredetected by threshold crossings of 500 µV occurring simultaneously on atleast 4 of 12 randomly selected channels. A 3.5 ms window of data aroundeach detected artifact was then removed and adjacent data segments wererejoined. This approach leaves the vast majority of the neural dataintact. A series of control experiments confirmed the removal of thestimulation artifact across several contexts. Data collecteddemonstrates the robust ability to remove artifacts from the data withthis approach prior to signal analysis.

Neural activity was next measured using MWP. Wavelet decomposition wasapplied to the raw voltage data, using the ‘db4’ mother wavelet and 11wavelet scales. Four wavelet scales 3-6 were used, corresponding to themultiunit frequency band spanning approximately 234 to 3,750 Hz. Themean of the wavelet coefficients for each scale of each channel wascalculated every 100 ms and a 1 s wide boxcar filter was applied tosmooth the data. Baseline drift in the data was estimated by using a 15s boxcar filter and was subtracted from the smoothed mean waveletcoefficients for the corresponding 100 ms window. The mean coefficientswere then standardized per channel, per scale, by subtracting the meanand dividing by the standard deviation of those scales and channelsduring the training blocks. The four scales were then combined byaveraging the standardized coefficients for each channel, resulting in96 MWP values, one for each electrode in the array, for every 100 ms ofdata. The resulting MWP values were used as input into the givennon-linear SVM decoders. The SVM model training and testing methods aredetailed below for both the passive sensory stimulation or active objectmanipulation experiments.

The embodiment of the FES device 20 used in the experiments to stimulatethe arm musculature and produce movement consisted of a multi-channelstimulator and a flexible cuff with up to 130 electrodes that is wrappedaround the participant’s forearm. During use, hydrogel disks (Axelgaard,Fallbrook, CA) were placed between the electrodes and skin to act as aconduction enhancer. The electrodes are 12 mm in diameter and werespaced at 22 mm intervals along the longitudinal axis of the forearm and15 mm intervals in the transverse direction. Current-controlled,monophasic rectangular pulses (50 Hz pulse rate and 500 µs pulse width)were used to provide electrical stimulation. Pulse amplitudes rangedfrom 0 to 20 mA and were updated every 100 ms. Stimulator calibrationswere performed for each movement using an anatomy-based trial-and-errormethod to determine appropriate electrode spatial patterns.

In passive sensory stimulation experiments, we first assessed evokedneural activity in left primary motor cortex M1 using bi-polarelectro-tactile stimulation at skin locations on the participant’s armand hand. Electro-tactile stimulation was chosen in part for its safetyand precise electronic control of stimulus features, and its ability toevoke activity in M1 following from pilot recordings. We targeted fourskin locations innervated by the spinal cord above, at, and below theparticipant’s C5/C6 SCI. The skin stimulation locations were: (1) C5dermatome (forearm; electrode location: skin above the extensor carpiradialis longus); (2) C6 dermatome (thumb; electrode location: skinabove the distal phalanx of digit 1); (3) C7 dermatome (index; electrodelocation: skin above the distal phalanx of digit 2); and (4) C7dermatome (middle; electrode location: skin above the distal phalanx ofdigit 3). The notation: “Forearm”, “thumb”, “index”, and “middle” areused to describe these four skin stimulation sites. A subset of controlrecordings were also performed on the opposite arm ipsilateral to the M1implant for the homotopic thumb and forearm locations. Cutaneouslandmarks and/or ink markings were used throughout as needed to confirmskin stimulation locations. The participant wore an eye mask and earplugs during all passive sensory stimulation experiments tosignificantly reduce any external visual and auditory events duringrecordings. Recordings were video recorded and performed under thesupervision of a licensed physiatrist.

The stimulation interface for a given skin location consisted of a pairof hydrogel disk electrodes adhered to a modified version of the FESinterface used in our previous studies. Each hydrogel disk electrode(Axelgaard, Fallbrook, CA) is 12 mm in diameter, 1.27 mm thick, spacedby ~2-3 mm, and attached to a metal electrode consisting of copper withan electroless nickel immersion gold coating embedded in the polyimideflex circuit. We used two current levels of stimulation: minimumintensity = 2.4 mA, and maximum intensity = 9.6 mA (current controlledstimulation, monophasic rectangular pulses, 50 Hz, 500 µs pulse width,100 ms train duration). Stimulation intensity was selected based on ourpilot studies to apply stimulation sufficient to evoke activity in M1(minimum intensity) and up to an intensity below a noxious level(maximum intensity). Fifty replicates of stimulation were performedwithin a given recording with an inter-stimulus interval of 2 s, toensure the relaxation of neural activity similar to our previousstudies. On a given recording day, 2 - 3 skin stimulation locations wererandomly selected for stimulation and simultaneous neural recordings.The amplitudes of current for a given skin stimulation location werealso randomly selected for a given location. Recordings were performedacross a total of ~5 months to assess the chronic viability of theevoked neural signal.

Experiments on evoked activity analyses are next described. InPeri-Stimulus Time Histograms (PSTHs) experiments, all neural recordingswere analyzed offline using MATLAB 2016b (The MathWorks). Followingstimulation artifact removal, the signal was band-passed filtered (3rdorder Butterworth filter; 300 - 3000 Hz). Single and multi-unit activitywas classified off-line using superparamagnetic clustering. Single unitswere in addition manually inspected, similar to previous studies. Foreach channel, the neural activity was first binned (20 ms bin width).PSTHs were then constructed for the neural data 1 second before andafter stimulation to assess evoked responses similar to previousstudies. A positive 95% confidence interval (C.I.) was then applied tothe given channel’s PSTH to identify significant neural modulation. Forchannels with a significant evoked response (i.e., a positive 95% C.I.crossing), we calculated the response magnitude (units: cumulativespikes above the 95% C.I., minimum of 3 consecutive bins needed) andresponse latency (the first significant bin, units = ms). The globalresponse magnitude was then estimated using the cumulative sum ofresponse magnitudes across array channels for the given condition. Wereport the global response magnitude (see FIG. 6C) and response latency(FIG. 6D).

Experiments on Local Field Potential Spectrograms and SpatiotemporalOrganization were also performed. The local field potential (LFP) wasfirst generated (3rd order Butterworth filter; 1 - 300 Hz). Spectrogramswere next constructed from the LFP to assess low frequency neuralmodulations during passive sensory stimuli (using ‘spectrogram’ inMATLAB). LFP from 5-20 Hz exhibited robust time-locked modulation duringsensory stimuli, corresponding to a well-studied sensorimotor frequencyband. The 5-20 Hz LFP band was therefore used for the following analyseson spatiotemporal organization of the evoked response. For each channel,all 5-20 Hz LFP oscillations were first extracted for analysis, andfurther classified as occurring during either REST (15 s period ofspontaneous activity, prior to stimulation) or 1 s after a sensorystimulus (POST). We separated the positive and negative phases of allLFP oscillations to assess the hyperpolarized and depolarized componentsof the oscillatory activity. The temporal order (i.e., rank) of LFP timeto peak was then calculated across all channels for a given recording toassess the spatiotemporal organization of the activity (score of 1:first channel active; through a score of 96: last channel active;modified from reference 40). The cumulative score across all stimuli wasthen converted to a normalized probability, corresponding to the givenchannel’s preferred order of activation. Each channel’s temporal orderprobability was assessed separately for REST and POST. Thisspatiotemporal analysis illustrates that stimuli evoke aspatiotemporally organized response across the recording array,potentially identifying a putative source of the evoked response.

To assess potential evoked traveling waves, we again used all 5-20 HzLFP oscillations for both positive and negative phases during the RESTand POST stimulus periods. We utilized LFP peak analyses, gradientmapping of the LFP vector representation, and subsequent directionalityof oscillatory activity, similar to previous studies on traveling wavesand wave-like propagating neural activity. For each extracted LFPoscillation, we generated a velocity vector map across the recordingarray, assessing the magnitude and direction of the activity for eachchannel. Next, we transferred each channel’s vector onto a phase plotfor directionality averaging and subsequent directional statistics.Finally, we computed the given oscillation’s circular variance and granddirectionality vector (i.e., resultant vector), averaged over all 96channels using the circular statistics toolbox for MATLAB. Oscillationswith low circular variance have a more prominent preferreddirectionality and spatial organization across the array, compared tooscillations with a higher circular variance. The circular variancevalues for all oscillations during REST and POST were then binned forthe given recording, to assess a potential effect of sensory stimulationon preferred directionality changes in the evoked response across time(20 ms bin width). Oscillations with a circular variance less than thenegative 95% confidence interval for the given recording were consideredtraveling waves. We report the grand directionality vector and thecorresponding traveling wave probability for all sensory recordings.

In experiments on decoding passive sensory stimulation, a nonlinear SVMclassifier was used to decode stimulus location for the passive sensorystimulation recordings at both a low and high stimulation intensity (SVMhyperparameters: γ = 0.001, C = 1). Two SVM models (low and highstimulation intensity) were each built with 5 classes: Rest, Forearm,Thumb, Index, and Middle. The input features for each model werecalculated as follows: (1) We recorded neural activity and calculatedMWP during ~350 total stimuli for each stimulus intensity and skinlocation across ~5 months; (2) MWP was standardized across blocks withineach day to account for day-to-day variability; (3) For each skinstimulation trial, defined by 0.2 s before and 0.8 s after a givensensory stimulus (this epoch was chosen due to the robust neuralmodulation that occurs during this time period around the stimulus, seeFIGS. 6B and 7A), MWP was vectorized to a 960-feature vector (96channels * 10 bins; where each bin spanned 100 ms); The same MWPvectorization process was applied to 350 randomly selected 1 s samplesof Rest data collected during spontaneous activity in the first 15 s ofa given recording; and (4) For each class, the vectorized MWP wasshuffled to remove any effect of stimulus order or recording time duringthe ~5-month period and equally assigned to either training or testingdata (~175 trials per class for training; ~175 trials per class fortesting). We report SVM model performance as a confusion matrix (seeFIGS. 8B and 8C; diagonal values = sensitivity; off-diagonal values =false positive rate). Sensitivity is calculated as the percentage ofcorrectly predicted class labels for the targeted class (true positiverate). False positive rate is calculated as the percentage ofincorrectly predicted class labels for a given off-target class. Rowsrepresent the actual recorded class, while the columns represent themodel’s predictions.

Some further aspects of the active object manipulation experiments arenext described. For clinical assessment of sensory function,monofilament testing was performed by a licensed physiatrist to evaluatethe participant’s hand sensory function (GRASSP assessment,Semmes-Weinstein monofilaments; Toronto, ON). The palmar and dorsalaspects of digits 1 (thumb), 2 (index), and 3 (middle) were exposed tomultiple trials of either 0.4, 2, 4, or 300 g of force while theparticipant was blind-folded. Trial location and force level wererandomized. The participant was asked to report the application of theapplied tactile stimulus. The following scores were generated toquantify the participant’s tactile acuity: 4 = 0.4 g detection at 66 %;3 = 2 g at 33 %; 2 = 4 g at 33 %; 1 = 300 g at 33 %; 0 = 300 g at 0 %.The participant uses his hand to manipulate objects during BCIoperation. We assessed the participant’s ability to recognize objecttouch during FES-mediated and grip (standardized objects tested from theAction Research Arm Test: small cylinder (1 cm diameter) and largecylinder (2 cm diameter)). The participant was again blind-folded, andthe object was placed between digits 1 and 2 without touching the skinon randomized cues where a grip was triggered (small cylinder: lateralpinch grip; large cylinder: can grip). The given grip was cued for aduration of 3 seconds. The participant then reported whether there wasan object in his hand. Each grip cue was bounded by rest cues withrandom durations between 5 to 6 seconds. We report Object TouchRecognition as the percentage of cues the participant correctlyidentified there was an object present during the grip for each object.

In experiments on decoding active touch, we trained SVM decoders torecognize active hand touch events in real-time. These decoders weretrained using neural data during active object touch, in contrast to thepassive sensory stimulation decoders described above (see DecodingPassive Sensory Stimulation). We used the can object, a part of thestandard clinical grasp and release test battery (5.4 × 9.1 cm). Formodel training, we recorded 9 total cues of labeled touch data, witheach cue consisting of a 6 second period. Cues were conveyed by avirtual hand on a computer monitor. Each cued period of touch data wasbounded by rest cues with random durations between 5 to 6 seconds. Foreach touch cue period, the participant first moved his hand down ontoand around the can object for 3 seconds, followed by a scripted objectgrip period for an additional 3 seconds where functional electricalstimulation (FES) triggered a more forceful grip. Therefore, touchdecoder model training consisted of neural data during: 1) movement ontothe object, 2) natural touch of the object, and finally 3) additionalFES mediated touch. This touch decoder model was then tested on 4 cuetypes to assess model performance during ‘touch’ and ‘no touch’ events.The participant completed the following cued events: (1) 3 seconds ofnatural touch of the object followed by 3 seconds FES mediated touch(‘Touch’), (2) 6 seconds of natural object touch (‘Touch’), (3) 6seconds of identical movement without the object present (‘No Touch’),(4) 6 seconds of FES without the object present (‘No Touch’). For thistouch decoder testing, we report model responsiveness during the 4 cuetypes, defined as the percentage of time the touch decoder output wasabove the activation threshold during the given cue (activationthreshold = 0.5). This touch decoder was then used to trigger theclosed-loop haptic feedback interface 34 during object manipulation. Ina subset of experiments, we also assessed touch decoder timing duringsimultaneous recording of applied force (force transducer interface:custom designed piezoresistive sensor pad (FlexiForce; Boston, MA)interfaced with an Arduino Mega 2560 board transferring force data tothe PC).

For decoding movement intention, we built movement decoders formanipulating the can object (a part of the grasp and release test).Briefly, the participant was prompted to imagine performing a can gripand movement, using a virtual hand displayed on a computer monitor. Eachmovement cue lasted 3-4 seconds, and was bounded by rest cues withrandom durations between 5 to 6 seconds. During a movement cue, FEStriggered the can grip. FES was controlled by the SVM movement decoderstarting after the first 3 movement cues. This motor decoder model wasupdated during subsequent training cues until a sufficiently accuratemodel was built (accuracy > ~80%). This motor decoder was then used tocontrol FES during object manipulation.

In experiments employing the haptic device 34, this embodiment of thehaptic feedback interface 34 consisted of 3 low-noise vibrotactile coinmotors affixed to a velcro band wrapped around the participant’s rightbicep (coin motor details: 12 mm diameter, 3.4 mm height, 2.6 G forceoutput; Need for Power; Shenzhen, Guangdong, China). This interface wastethered to an Arduino Mega 2560 board to power and control vibrotactilehaptic feedback. Haptic feedback interfaces targeting the skin over thebiceps have been used in several sensory feedback studies and is wellstudied. Our pilot data confirm that the participant’s right bicepexhibited normal sensory function, and haptic stimulation was recognizedon 100% stimuli. The interface was designed to ensure participantcomfort during movement. The vibrotactile motors achieved maximumamplitude (2.6 G) within 1 ms of controller signal initiation. Allhaptic feedback interface communication was also recorded. This hapticfeedback interface was controlled by the touch decoders outlined aboveand was triggered in real time during closed-loop sensory feedback tasksthat employed the haptic device 34.

Experiments investigating functional improvement with and withoutclosed-loop haptic feedback were performed as follows. We assessed upperlimb function across a battery of four clinical assays under thesupervision of a licensed physiatrist. The haptic feedback interface 34was placed on the participant’s right bicep during all assessments, andfunction was assessed across trials during either a ‘no haptics’ or‘haptics’ condition. The ‘no haptics’ condition consisted of functionaltesting without any vibrotactile sensory feedback. ‘Haptics’ consistedof on demand touch decoder controlled closed-loop sensory feedback forrapidly conveying hand touch events back to the user. The participantwas blinded to the sensory feedback condition before a series ofassessment trials. All clinical assays were performed across 2 clinicaltesting days. The first clinical assessment was an extension of themonofilament testing. A touch decoder was first constructed aspreviously described. The palmar and dorsal aspects of digits 1 - 5 wereexposed to multiple trials of either 0.4, 2, 4, or 300 g of force whilethe participant was blind-folded. Trial location and force level wererandomized. The participant was asked to report the application of theapplied tactile stimulus. During the ‘haptics’ condition, the touchdecoder controlled haptic feedback. Haptic feedback time locked tomechanical stimulation was reported by the participant and constituted apositive report of skin stimulation. All trials were recorded with highspeed video for offline analysis.

The second clinical assessment was an extension of the Object TouchRecognition test. The standardized large cylinder object was used. Atouch decoder was first constructed as previously described. Theparticipant was again blind-folded, and the object was placed betweendigits 1 and 2 without touching the skin on randomized cues where a gripwas triggered by FES during shuffled series of ‘haptics’ or ‘no haptics’conditions. Grip was cued for a duration of 3 seconds. The participantthen reported whether there was an object in his hand. Each grip cue wasbounded by rest cues with random durations between 5 to 6 seconds. Weagain report Object Touch Recognition as the percentage of cues theparticipant correctly identified there was an object present duringgrip.

The third clinical assessment consisted of the modified grasp andrelease test (GRT) only using the can object. A touch decoder andmovement decoder was constructed as previously described. Theparticipant was then cued to repeatedly grasp, move, and release theobject during shuffled series of ‘haptics’ or ‘no haptics’ conditiontrials. After each GRT trial, the participant reported his sense ofagency (SoA) (i.e., “How in control did you feel of the movement andgrip?”). The SoA score ranged from 0-100, similar to previous studies (0= poor sense of control; 100 = perfect sense of control).

The fourth clinical assessment was a modified GRT again using only thecan object. A touch decoder and movement decoder was constructed aspreviously described. The participant was instructed to grasp, transfer,and release the can object onto an elevated platform as fast as possiblerepeatedly during shuffled series of ‘haptics’ or ‘no haptics’ conditiontrials. Each GRT assessment period consisted of two 60 second movementperiods separated by a 20 second rest period. All GRT trials wererecorded with high speed video for offline analysis. We report thenumber of objects successfully transferred during the movement periods.We also assessed the interval between the touch decoder and movementdecoder to examine the neurophysiological substrates of GRT performancewith and without haptic feedback (high speed video was also used inaddition to decoder times to confirm touch and movement event starttimes). The touch or movement event start time was calculated across GRTtrials using the time each decoder crossed the significance threshold totrigger the respective assistive device (decoders were normalized from0 - 1; significance threshold = 0.5). We report the interval (s) betweenthe touch and movement decoder.

Some further aspects of the data analysis and statistics are as follows.Normality tests were performed for each analysis to determine ifparametric or nonparametric statistics should be used. All statisticaltests were two-tailed and performed in MATLAB 2016b. An alpha level of0.05 was accepted for significance unless Bonferroni corrections arenoted. Effects of sensory stimuli on evoked M1 neural activity wereevaluated using separate one-way ANOVAs for the minimum and maximumstimulation intensities. The factor was skin location with 4 levels:forearm, thumb, index, and middle. Tukey’s post-hoc test was used todetermine differences in response magnitude and response latency acrossskin locations (see FIGS. 6C and 6D). Effects of skin stimulation onspatiotemporal response organization were evaluated using independentsamples t-tests for both the response latency topography and travelingwave probability data. Effects of skin stimulation on preferreddirectionality vector magnitudes was assessed using circular statistics.

A one-sided t-test was used to determine if decoder performance valueswere above chance for the passive sensory stimulation data (confusionmatrices, see FIGS. 8B and 8C). Each confusion matrix value was comparedto a chance prediction level for statistical evaluation. Chance levelswere generated by randomly permuting the data labels 10 times. Valuesfrom unrandomized label permutation were then compared with values fromrandomized label permutation. A Bonferroni corrected alpha value of0.002 was used for significance (0.05 / 25 comparisons).

For the active object manipulation experiments, touch decoderresponsiveness values were assessed using a one-way ANOVA. The factorwas cue type with four levels: Object Touch & FES, Object Touch, FESalone, and Movement alone. Tukey’s post-hoc test was used to determinedifferences in touch decoder responsiveness across cue type. Functionalimprovement assessments were performed across two separate clinicaltesting days for the following total trial counts: object touchrecognition: 36 trials (for either ‘haptics’ and ‘no haptics’), SoA: 24trials (for either ‘haptics’ and ‘no haptics’), GRT performance &decoder interval: 77 trials (‘haptics’) and 73 trials (‘no haptics’).Effects of closed-loop haptic feedback were assessed using independentsamples t-tests for the object touch recognition, SoA, GRT performance,and decoder interval data, comparing the ‘no haptics’ to ‘haptics’conditions.

The preferred embodiments have been illustrated and described.Modifications and alterations will occur to others upon reading andunderstanding the preceding detailed description. It is intended thatthe present disclosure be construed as including all such modificationsand alterations insofar as they come within the scope of the appendedclaims or the equivalents thereof.

1. An apparatus for assisting a patient having a spinal cord injury, theapparatus comprising: an electrical brain signal monitoring interfaceconfigured to record at least one electrical brain signal of thepatient; a functional electrical stimulation (FES) device configured toconnect via FES electrodes with a paralyzed portion of the patient thatis paralyzed due to the spinal cord injury and to control the paralyzedportion of the patient by applying FES to the paralyzed portion of thepatient via the FES electrodes; an electronic processor operativelyconnected with the electrical brain signal monitoring interface toreceive the at least one electrical brain signal and with the FES deviceto control the FES applied to the paralyzed portion of the patient; anda non-transitory storage medium storing instructions readable andexecutable by the electronic processor, the instructions including:electrical brain signal demultiplexing instructions that are executableby the electronic processor to demultiplex the at least one electricalbrain signal into an efferent motor intention signal and at least oneafferent sensory signal; and FES control instructions that areexecutable by the electronic processor to control the FES applied to theparalyzed portion of the patient by the FES device based on at least theefferent motor intention signal.
 2. The apparatus of claim 1 wherein theelectrical brain signal demultiplexing instructions are executable bythe electronic processor to demultiplex the at least one electricalbrain signal into the efferent motor intention signal and the at leastone afferent sensory signal including at least an afferentproprioception sense signal generated by a proprioception sense of theparalyzed portion of the patient.
 3. The apparatus of claim 2 whereinthe FES control instructions are executable by the electronic processorto control the FES applied to the paralyzed portion of the patient bythe FES device based on at least the efferent motor intention signal andthe afferent proprioception sense signal.
 4. The apparatus of claim 2wherein the paralyzed portion of the patient includes a hand, and theFES control instructions are executable by the electronic processor to:determine an orientation of a palm or wrist of the patient based on theafferent proprioception sense signal; and control the FES applied toorient the hand to grasp an object based on the determined orientationand then control the FES to cause the hand to grasp the object.
 5. Theapparatus of claim 1 wherein the FES control instructions are executableby the electronic processor to control the FES applied to the paralyzedportion of the patient by the FES device based on the efferent motorintention signal and not based on the at least one afferent sensorysignal.
 6. The apparatus of claim 1 wherein the FES device comprises anFES cuff configured to connect via FES electrodes with a paralyzed arm,hand, leg, or foot of the patient that is paralyzed due to the spinalcord injury and to control the paralyzed arm, hand, leg, or foot of thepatient by applying FES to the paralyzed arm, hand, leg, or foot via theFES electrodes.
 7. The apparatus of claim 1 wherein the electrical brainsignal demultiplexing instructions are executable by the electronicprocessor to apply a machine learning component to demultiplex the atleast one electrical brain signal into the efferent motor intentionsignal and the at least one afferent sensory signal.
 8. A non-transitorystorage medium storing instructions readable and executable by anelectronic processor that is operatively connected to receive at leastone electrical brain signal from a patient and to operate a functionalelectrical stimulation (FES) device to apply FES to control a paralyzedportion of the patient that is paralyzed due to a spinal cord injury ofthe patient, the stored instructions including: electrical brain signaldemultiplexing instructions that are executable by the electronicprocessor to demultiplex the at least one electrical brain signal intoan efferent motor intention signal and at least one afferent sensorysignal; and FES control instructions that are executable by theelectronic processor to control the FES applied to the paralyzed portionof the patient by the FES device based on at least the efferent motorintention signal.
 9. The non-transitory storage medium of claim 8wherein the electrical brain signal demultiplexing instructions areexecutable by the electronic processor to demultiplex the at least oneelectrical brain signal into the efferent motor intention signal and theat least one afferent sensory signal including at least an afferentproprioception sense signal generated by a proprioception sense of theparalyzed portion of the patient.
 10. The non-transitory storage mediumof claim 9 wherein the FES control instructions are executable by theelectronic processor to control the FES applied to the paralyzed portionof the patient by the FES device based on at least the efferent motorintention signal and the afferent proprioception sense signal.
 11. Thenon-transitory storage medium of claim 9 wherein the paralyzed portionof the patient includes a hand, and the FES control instructions areexecutable by the electronic processor to: determine an orientation of apalm or wrist of the patient based on the afferent proprioception sensesignal; and control the FES applied to orient the hand to grasp anobject based on the determined orientation and then control the FES tocause the hand to grasp the object.
 12. The non-transitory storagemedium of claim 8 wherein the FES control instructions are executable bythe electronic processor to control the FES applied to the paralyzedportion of the patient by the FES device based on the efferent motorintention signal and not based on the at least one afferent sensorysignal.
 13. The non-transitory storage medium of claim 8 wherein theelectrical brain signal demultiplexing instructions are executable bythe electronic processor to apply a machine learning component todemultiplex the at least one electrical brain signal into the efferentmotor intention signal and the at least one afferent sensory signal. 14.A method comprising: receiving at least one electrical brain signal froma patient; by an electronic processor, demultiplexing the at least oneelectrical brain signal into an efferent motor intention signal and atleast one afferent sensory signal; and by the electronic processor,controlling a functional electrical stimulation (FES) device to applyFES to control a paralyzed portion of the patient that is paralyzed dueto a spinal cord injury of the patient; wherein the controlling of theFES device is based on at least the efferent motor intention signal. 15.The method of claim 14 wherein the demultiplexing comprisesdemultiplexing the at least one electrical brain signal into theefferent motor intention signal and the at least one afferent sensorysignal including at least an afferent proprioception sense signalgenerated by a proprioception sense of the paralyzed portion of thepatient.
 16. The method of claim 15 wherein the controlling of the FESdevice is based on at least the efferent motor intention signal and theafferent proprioception sense signal.
 17. The method of claim 15 whereinthe paralyzed portion of the patient includes a hand, and thecontrolling of the FES device includes: determining an orientation of apalm or wrist of the patient based on the afferent proprioception sensesignal; and controlling the FES applied to orient the hand to grasp anobject based on the determined orientation and then control the FES tocause the hand to grasp the object.
 18. The method of claim 14 whereinthe controlling of the FES device is based on the efferent motorintention signal and is not based on the at least one afferent sensorysignal.